Towards Effective Low-bitwidth Convolutional Neural Networks

In this work, we aims to effectively train convolutional neural networks with both low-bitwidth weights and low-bitwidth activations. Optimization of a lowprecision network is typically extremely unstable and it is easily trapped in a bad local minima, which results in noticeable accuracy loss. To mitigate this problem, we propose two novel approaches. On one hand, unlike previous methods that quantize weights and activations simultaneously, we instead propose to use an optimization strategy to progressively finding good local minima. On the other hand, we distill the knowledge from the full-precision model to improve the training process and the final performance of the low-precision model. Furthermore, we can learn our low-precision model from scratch or fine-tune from the full-precision model without losing flexibility. Extensive experiments on various datasets( i.e., CIFAR-100 and ImageNet) show that the proposed method shows minimum loss in accuracy ( i.e., at 2-bit and 4-bit quantization) and sometimes even improves the performance compared to its full-precision counterpart on popular network architectures ( i.e., AlexNet and ResNet-50).